Low-power, fast-response machine learning autofocus enhancements

ABSTRACT

Computing devices, such as mobile computing devices, have access to one or more image sensors that can capture images with multiple subjects. Some of these subjects may be known to the user capturing an image with the image sensor. The user may prefer to have the captured image data be optimized around the known subjects. Low-power, fast-response machine learning logic can be configured to allow for the generation of a plurality of inference data. This inference data can be utilized along with other sensor data, such as a motion sensor, for the generation of one or more image sensor configuration changes that may be implemented to optimize the subsequent capture of image data. This cycle of image data analysis, image sensor optimization, and subsequent capture can continue multiple times until a threshold of optimization or time is met. The captured image data optimized around the known subjects is then stored.

PRIORITY

This application is a continuation of and claims the benefit of priorityto U.S. patent application Ser. No. 17/360,373, filed Jun. 28, 2021,which is incorporated in its entirety herein.

FIELD

The present disclosure relates to image processing. More particularly,the present disclosure relates to utilizing low-power, artificialintelligence-based systems to provide fast-response autofocusenhancements for capturing optimized images.

BACKGROUND

As technology has grown over the last decade, the growth of image datasuch as photographic content has increased dramatically, especially withthe advent of mobile computing devices with built-in image sensors. Thisincrease in image data has generated a greater demand for automaticclassification and optimization, especially as image data is capturedwith one or more subjects known to the person capturing the image. Inresponse, neural networks and other artificial intelligence methods havebeen increasingly utilized to generate automatic classifications,detections, and other optimizations.

However, as image data and the neural networks used to analyze them haveincreased in size and complexity, a higher computational and powerdemand is created. More data to process requires more time to processall of the data. Likewise, more complex neural networks require moreprocessing power to parse the data. Traditional methods of handlingthese problems include trading a decrease in output accuracy forincreased processing speed, or conversely, increasing the outputaccuracy for a decrease in processing speed.

SUMMARY

Systems and methods for optimizing image data utilizing low-power,fast-response machine learning logic in accordance with embodiments ofthe invention are disclosed herein. In many embodiments, a deviceincludes an image sensor, a Non-Volatile Memory (NVM) for storing dataand executable instructions, and a processor communicatively coupled tothe NVM. The processor can be configured to direct the device to receiveimage data for processing and pass the received image data to a machinelearning model which can recognize a known subject within the imagedata, generate a plurality of inferences, and utilize the plurality ofinferences to generate image optimization data. The image optimizationdata includes one or more image sensor configurations for optimizing anarea within the image data associated with the known subject duringsubsequent image data captures.

In further embodiments, the processor includes a machine learningprocessor itself comprising a plurality of non-volatile memory cells tostore weights for the machine learning model, and wherein the machinelearning processor is configured to apply signals corresponding to thereceived image data, via one or more signal lines associated with thememory cells, to the memory cells, to generate the plurality ofinferences.

In more embodiments, the non-volatile memory cells are Spin-Orbit TorqueMagnetoresistive Random-Access Memory (SOT MRAM) memory cells.

In additional embodiments, the device is also configured with a motionsensor to receive motion data from the motion sensor and generate theimage optimization data based on the plurality of inferences and thereceived motion data.

In still further embodiments, the device is configured to continuecapturing image data until the optimization of the known subject withinthe image data exceeds a predetermined threshold.

In still additional embodiments, the image data is captured in discreteunits.

In yet further embodiments, the device generates unique imageoptimization data between each discrete unit of image data captures.

In a series of embodiments, the predetermined threshold is related tothe focus of the known subject within the image data.

In various embodiments, the device is also configured to recognize twoor more known subjects within the image data and generate imageoptimization data including one or more image sensor configurations foroptimizing areas of the image data associated with the two or more knownsubjects.

In a number of embodiments, the area associated with the known subjectis a bounding box encasing the known subject.

In more embodiments, the area associated with the known subject is amask covering the known subject.

In still more embodiments, a method for generating machine-learningbased image optimization data includes receiving image data and passingthe received image data to a low-power, fast-response machine learningmodel. Then, the model can recognize one or more known subjects withinthe image data, determine a known subject area for optimization based onthe one or more known subjects, and generate a plurality of inferencesvia the machine learning model. The device can further utilize theplurality of inferences to generate image optimization data includingone or more image sensor configurations for optimizing the known subjectarea within the image data, providing the image optimization data to animage sensor for capturing subsequent image data, and processingsubsequently received image data until a predetermined optimizationthreshold is exceeded.

In further additional embodiments, the optimization includes improvingthe focus of the known subject area.

In still more embodiments, the low-power, fast-response machine learningmodel is executed in a plurality of Magnetoresistive Random-AccessMemory (MRAM) based machine learning devices.

In another series of embodiments, the generation of the plurality ofinferences is completed in less than one millisecond.

In certain embodiments, a device, includes a Non-Volatile Memory (NVM)for storing data and executable instructions, and a processorcommunicatively coupled to the NVM. The processor can be configured todirect the device to receive image data for processing, pass thereceived image data to a machine learning model, which can thenrecognize a known subject within the image data, generate a plurality ofinferences, utilize the plurality of inferences to generate imageoptimization data. The image optimization data can include one or moreimage sensor configurations for optimizing image data capture areasassociated with the known subject.

In yet more embodiments, the processor includes a machine learningprocessor comprising a plurality of non-volatile memory cells to storeweights for the machine learning model, and wherein the machine learningprocessor is configured to apply signals corresponding to the receivedimage data, via one or more signal lines associated with the memorycells, to the memory cells, to generate the plurality of inferences.

In more various embodiments, the image sensor includes a plurality ofvarying focal length image sensors.

In additional embodiments again, the image sensor also includes a LightDetection and Ranging (LiDAR) camera

In still yet further embodiments, an image sensor array is disposedwithin a mobile computing device.

In yet additional embodiments, the known subject is selected via aninput received by the mobile computing device.

Although the description above contains many specificities, these shouldnot be construed as limiting the scope of the invention but as merelyproviding illustrations of some of the presently preferred embodimentsof the invention. Various other embodiments are possible within itsscope. Accordingly, the scope of the invention should be determined notby the embodiments illustrated, but by the appended claims and theirequivalents.

BRIEF DESCRIPTION OF DRAWINGS

The above, and other, aspects, features, and advantages of severalembodiments of the present disclosure will be more apparent from thefollowing description as presented in conjunction with the followingseveral figures of the drawings.

FIG. 1A is conceptual diagram of an artificial neural network inaccordance with an embodiment of the disclosure;

FIG. 1B depicts a matrix-vector multiplication operation of theartificial neural network of FIG. 1A in accordance with an embodiment ofthe disclosure;

FIG. 1C is an example cross-point memory array suitable for performingthe matrix-vector multiplication operation depicted in FIG. 1B inaccordance with various embodiments of the disclosure is shown inaccordance with an embodiment of the disclosure;

FIG. 2 is conceptual cross-point memory array that may be utilizedwithin a low-power fast-response machine learning system according tovarious embodiments of the disclosure;

FIG. 3 depicts the conceptual process of generating optimized image datain accordance with an embodiment of the disclosure;

FIG. 4 is a conceptual schematic diagram of a device suitable forprocessing optimized image data utilizing low-power fast-responsemachine learning models in accordance with an embodiment of thedisclosure;

FIG. 5A is a conceptual illustration of an image with a known subjectprocessed with a bounding box in accordance with an embodiment of thedisclosure;

FIG. 5B is a conceptual illustration of an image with a known subjectprocessed with a pixel mask in accordance with an embodiment of thedisclosure;

FIG. 6 is a conceptual illustration of an image processed with multipleknown subjects in accordance with an embodiment of the disclosure;

FIG. 7 . is a flowchart depicting a process for optimizing image datawith a plurality of known subjects in accordance with an embodiment ofthe disclosure; and

FIG. 8 is a flowchart depicting a process for utilizing fast-responselow-power machine learning logic to optimize received image data inaccordance with an embodiment of the disclosure.

Corresponding reference characters indicate corresponding componentsthroughout the several figures of the drawings. Elements in the severalfigures are illustrated for simplicity and clarity and have notnecessarily been drawn to scale. For example, the dimensions of some ofthe elements in the figures might be emphasized relative to otherelements for facilitating understanding of the various presentlydisclosed embodiments. In addition, common, but well-understood,elements that are useful or necessary in a commercially feasibleembodiment are often not depicted in order to facilitate a lessobstructed view of these various embodiments of the present disclosure.

DETAILED DESCRIPTION

In response to the problems described above, devices and methods arediscussed herein that optimize known subjects within captured image dataduring the image capture process. As those skilled in the art willrecognize, many computing devices, including mobile computing devicesinclude one or more image sensors for capturing image data. For example,a mobile phone can have an image sensor array that can be used to takephotographs selected by a user. The user may know or have a preferencefor one or more subjects within the image captured by the image data.Conventional image capturing processes may attempt to optimize the imagedata as a whole or on a predetermined area. However, when presented withmultiple subjects, conventional methods of image data optimization donot discern one subject from another insofar as a subject within theimage data may be known to a user.

A user may attempt to capture image data associated with an eventwherein a number of subjects are captured. For example, a user may takea photograph of their child playing in a soccer game with otherchildren. In many instances, a user may prefer that the image datacaptured be optimized the area associated with their child, even at theexpense of optimization of other children if needed. The methods ofoptimizing image data being captured typically utilizes machine learninglogic (i.e., artificial neural networks) to process the data in asufficient speed.

However, traditional methods of machine learning logic require both toomuch time to process captured image data in time to adjust and optimizefurther image data capture during an image data capture cycle (i.e., thetime to take a photograph), and utilize levels of power that becomeprohibitive for use in mobile computing devices. However, as describedin more detail below, specialized low-power, fast-response machinelearning logic can be utilized to allow for generating a plurality ofinferences that can further be used for optimizing image data during thecapture process. Specifically, a device utilizing this low-power,fast-response machine learning logic can detect multiple subjects withinreceived image data, discern whether one or more of those subjects isknown to the user, and then optimize the image capture process aroundthe area of the image data associated with the one or more knownsubjects.

In some embodiments, one or more sensors available to the device, suchas for example a motion sensor, can also be utilized along with theplurality of inferences to generate optimization data for adjusting animage sensor for capturing more optimized image data. This cycle ofinference generation, optimization changes, and subsequent image datacapture and evaluation can continue a number of times until apredetermined number of cycles has occurred or a threshold level ofoptimization has been achieved. The determination of when to end theimage capture and optimization cycle can also occur upon the change ofimage data. For example, if the user taking the image moves their mobilecomputing device away from the subjects, the moment being captured maybe over and the large change in data between image capture cycles can bedetected and utilized to determine that the current image capture andoptimization cycle should end.

The determination of known subjects within the device can occur manuallyfrom the user, or from other data mining methods. For example, a mobilecomputing device with an available image sensor may have one or morepreviously captured images stored within the memory of the device. Thosepreviously captured images may be processed to determine a number ofsubjects that occur a predetermined number of times within the imageset. In other embodiments, the previously captured images may alreadyhave known subject data that may be utilized or shared to select the oneor more known subjects in future image data capture cycles.

Aspects of the present disclosure may be embodied as an apparatus,system, method, or computer program product. Accordingly, aspects of thepresent disclosure may take the form of an entirely hardware embodiment,an entirely software embodiment (including firmware, resident software,micro-code, or the like) or an embodiment combining software andhardware aspects that may all generally be referred to herein as a“function,” “module,” “apparatus,” or “system.” Furthermore, aspects ofthe present disclosure may take the form of a computer program productembodied in one or more non-transitory computer-readable storage mediastoring computer-readable and/or executable program code. Many of thefunctional units described in this specification have been labeled asfunctions, in order to emphasize their implementation independence moreparticularly. For example, a function may be implemented as a hardwarecircuit comprising custom VLSI circuits or gate arrays, off-the-shelfsemiconductors such as logic chips, transistors, or other discretecomponents. A function may also be implemented in programmable hardwaredevices such as via field programmable gate arrays, programmable arraylogic, programmable logic devices, or the like.

Functions may also be implemented at least partially in software forexecution by various types of processors. An identified function ofexecutable code may, for instance, comprise one or more physical orlogical blocks of computer instructions that may, for instance, beorganized as an object, procedure, or function. Nevertheless, theexecutables of an identified function need not be physically locatedtogether but may comprise disparate instructions stored in differentlocations which, when joined logically together, comprise the functionand achieve the stated purpose for the function.

Indeed, a function of executable code may include a single instruction,or many instructions, and may even be distributed over several differentcode segments, among different programs, across several storage devices,or the like. Where a function or portions of a function are implementedin software, the software portions may be stored on one or morecomputer-readable and/or executable storage media. Any combination ofone or more computer-readable storage media may be utilized. Acomputer-readable storage medium may include, for example, but notlimited to, an electronic, magnetic, optical, electromagnetic, infrared,or semiconductor system, apparatus, or device, or any suitablecombination of the foregoing, but would not include propagating signals.In the context of this document, a computer readable and/or executablestorage medium may be any tangible and/or non-transitory medium that maycontain or store a program for use by or in connection with aninstruction execution system, apparatus, processor, or device.

Computer program code for carrying out operations for aspects of thepresent disclosure may be written in any combination of one or moreprogramming languages, including an object-oriented programming languagesuch as Python, Java, Smalltalk, C++, C#, Objective C, or the like,conventional procedural programming languages, such as the “C”programming language, scripting programming languages, and/or othersimilar programming languages. The program code may execute partly orentirely on one or more of a user's computer and/or on a remote computeror server over a data network or the like.

A component, as used herein, comprises a tangible, physical,non-transitory device. For example, a component may be implemented as ahardware logic circuit comprising custom VLSI circuits, gate arrays, orother integrated circuits; off-the-shelf semiconductors such as logicchips, transistors, or other discrete devices; and/or other mechanicalor electrical devices. A component may also be implemented inprogrammable hardware devices such as field programmable gate arrays,programmable array logic, programmable logic devices, or the like. Acomponent may comprise one or more silicon integrated circuit devices(e.g., chips, die, die planes, packages) or other discrete electricaldevices, in electrical communication with one or more other componentsthrough electrical lines of a printed circuit board (PCB) or the like.Each of the functions and/or modules described herein, in certainembodiments, may alternatively be embodied by or implemented as acomponent.

A circuit, as used herein, comprises a set of one or more electricaland/or electronic components providing one or more pathways forelectrical current. In certain embodiments, a circuit may include areturn pathway for electrical current, so that the circuit is a closedloop. In another embodiment, however, a set of components that does notinclude a return pathway for electrical current may be referred to as acircuit (e.g., an open loop). For example, an integrated circuit may bereferred to as a circuit regardless of whether the integrated circuit iscoupled to ground (as a return pathway for electrical current) or not.In various embodiments, a circuit may include a portion of an integratedcircuit, an integrated circuit, a set of integrated circuits, a set ofnon-integrated electrical and/or electrical components with or withoutintegrated circuit devices, or the like. In one embodiment, a circuitmay include custom VLSI circuits, gate arrays, logic circuits, or otherintegrated circuits; off-the-shelf semiconductors such as logic chips,transistors, or other discrete devices; and/or other mechanical orelectrical devices. A circuit may also be implemented as a synthesizedcircuit in a programmable hardware device such as field programmablegate array, programmable array logic, programmable logic device, or thelike (e.g., as firmware, a netlist, or the like). A circuit may compriseone or more silicon integrated circuit devices (e.g., chips, die, dieplanes, packages) or other discrete electrical devices, in electricalcommunication with one or more other components through electrical linesof a printed circuit board (PCB) or the like. Each of the functionsand/or modules described herein, in certain embodiments, may be embodiedby or implemented as a circuit.

Reference throughout this specification to “one embodiment,” “anembodiment,” or similar language means that a particular feature,structure, or characteristic described in connection with the embodimentis included in at least one embodiment of the present disclosure. Thus,appearances of the phrases “in one embodiment,” “in an embodiment,” andsimilar language throughout this specification may, but do notnecessarily, all refer to the same embodiment, but mean “one or more butnot all embodiments” unless expressly specified otherwise. The terms“including,” “comprising,” “having,” and variations thereof mean“including but not limited to”, unless expressly specified otherwise. Anenumerated listing of items does not imply that any or all of the itemsare mutually exclusive and/or mutually inclusive, unless expresslyspecified otherwise. The terms “a,” “an,” and “the” also refer to “oneor more” unless expressly specified otherwise.

Further, as used herein, reference to reading, writing, storing,buffering, and/or transferring data can include the entirety of thedata, a portion of the data, a set of the data, and/or a subset of thedata. Likewise, reference to reading, writing, storing, buffering,and/or transferring non-host data can include the entirety of thenon-host data, a portion of the non-host data, a set of the non-hostdata, and/or a subset of the non-host data.

Lastly, the terms “or” and “and/or” as used herein are to be interpretedas inclusive or meaning any one or any combination. Therefore, “A, B orC” or “A, B and/or C” mean “any of the following: A; B; C; A and B; Aand C; B and C; A, B and C.” An exception to this definition will occuronly when a combination of elements, functions, steps, or acts are insome way inherently mutually exclusive.

Aspects of the present disclosure are described below with reference toschematic flowchart diagrams and/or schematic block diagrams of methods,apparatuses, systems, and computer program products according toembodiments of the disclosure. It will be understood that each block ofthe schematic flowchart diagrams and/or schematic block diagrams, andcombinations of blocks in the schematic flowchart diagrams and/orschematic block diagrams, can be implemented by computer programinstructions. These computer program instructions may be provided to aprocessor of a computer or other programmable data processing apparatusto produce a machine, such that the instructions, which execute via theprocessor or other programmable data processing apparatus, create meansfor implementing the functions and/or acts specified in the schematicflowchart diagrams and/or schematic block diagrams block or blocks.

It should also be noted that, in some alternative implementations, thefunctions noted in the block may occur out of the order noted in thefigures. For example, two blocks shown in succession may, in fact, beexecuted substantially concurrently, or the blocks may sometimes beexecuted in the reverse order, depending upon the functionalityinvolved. Other steps and methods may be conceived that are equivalentin function, logic, or effect to one or more blocks, or portionsthereof, of the illustrated figures. Although various arrow types andline types may be employed in the flowchart and/or block diagrams, theyare understood not to limit the scope of the corresponding embodiments.For instance, an arrow may indicate a waiting or monitoring period ofunspecified duration between enumerated steps of the depictedembodiment.

In the following detailed description, reference is made to theaccompanying drawings, which form a part thereof. The foregoing summaryis illustrative only and is not intended to be in any way limiting. Inaddition to the illustrative aspects, embodiments, and featuresdescribed above, further aspects, embodiments, and features will becomeapparent by reference to the drawings and the following detaileddescription. The description of elements in each figure may refer toelements of proceeding figures. Like numbers may refer to like elementsin the figures, including alternate embodiments of like elements.

FIG. 1A depicts a conceptual example of an artificial neural network 100that includes input neurons x₁, x₂, x₃, . . . , xn, output neurons y₁,y₂, y₃, . . . , y_(m), and synapses 102 that connect input neurons x₁,x₂, x₃, . . . , xn to output neurons y₁, y₂, y₃, . . . , y_(m). In anembodiment, each synapse 102 has a corresponding weight w11, w12, w13, .. . , w_(nm).

In an embodiment, each input neuron x1, x2, x3, . . . , xn has anassociated value, each output neuron y₁, y₂, y₃, . . . , y_(m) has anassociated value, and each weight w₁₁, w₁₂, w₁₃, . . . , w_(nm) has anassociated value. The value of each output neuron y₁, y₂, y₃, . . . ,y_(m) may be determined as follows:

$\begin{matrix}{{y_{k} = {\sum\limits_{j = 1}^{n}{x_{j}w_{kj}}}},{k = 1},2,\ldots,m} & (1)\end{matrix}$

In matrix notation, equation (1) may be written as y=x^(T) W, where y isan m-element output vector, x is an n-element input vector, and W is ann×m array of weights, as depicted in FIG. 1B.

The matrix-vector multiplication operation depicted in FIG. 1B may beimplemented by utilizing multiplication and accumulation operations, inwhich each output neuron y₁, y₂, y₃, . . . , y_(m) has an associatedvalue equal to the sum of products of each input neuron x₁, x₂, x₃, . .. , with the corresponding weight w₁₁, w₁₂, w₁₃, . . . , wnm thatconnects each respective input neuron x₁, x₂, x₃, . . . , x_(n) to theoutput neuron y₁, y₂, y₃, . . . , y_(m).

In a number of embodiments, a cross-point memory array can be used toperform the multiplication and accumulation operations described above.Referring to FIG. 1C, an example cross-point memory array suitable forperforming the matrix-vector multiplication operation in accordance withvarious embodiments of the disclosure is shown. In many embodiments, thecross-point memory array 110 that may be utilized to performmatrix-vector multiplication operations such as those depicted in FIG.1B.

In various embodiments, the cross-point memory array 110 may include nrows and m columns of nodes 112 ₁₁, 112 ₁₂, . . . , 112 ₃₄. Each row ofthese nodes 112 ₁₁, 112 ₁₂, . . . , 112 ₃₄ can be coupled to one of nfirst conductive lines (e.g., word lines (WL1, WL2, WL3, WL4).Additionally, each column of nodes 112 ₁₁, 112 ₁₂, . . . , 112 ₃₄ iscoupled to one of m second conductive lines (e.g., bit lines BL1, BL2,BL3). Those skilled in the art will understand that cross-point memoryarrays may include more or fewer than four word lines, and as well asfewer than three bit lines, and can have more or fewer than twelve nodesas depicted herein.

In certain embodiments, each node 112 ₁₁, 112 ₁₂, . . . , 112 ₃₄ of across-point memory array 110 may include a non-volatile memory cellhaving an adjustable resistance. In further embodiments, thenon-volatile memory cells in nodes 112 ₁₁, 112 ₁₂, . . . , 112 ₃₄ may beprogrammed to store a corresponding weight of an n×m array of weightsw₁₁, w₁₂, w₁₃, w₃₄, respectively. Thus, each node 112 ₁₁, 112 ₁₂, . . ., 112 ₃₄ is labeled with a corresponding weight w₁₁, w₁₂, w₁₃, . . . ,w₃₄, respectively, programmed in the corresponding non-volatile memorycell of the node. In an embodiment, each weight w₁₁, w₁₂, w₁₃, w₃₄corresponds to a conductance of the non-volatile memory cell in eachnode 112 ₁₁, 112 ₁₂, . . . , 112 ₃₄, respectively. The weights may beprogrammed, for example, during a training phase of the neural network.A common training method involves the weights being selectively and/oriteratively updated using an algorithm such as, but not limited to, backpropagation.

Input voltages Vin₁, Vin₂, Vin₃ and Vin₄ are shown applied to word linesWL1, WL2, WL3, WL4, respectively. The magnitudes of input voltages Vin₁,Vin₂, Vin₃ and Vin₄ can correspond to the associated values of inputneurons x₁, x₂, x₃ and x₄, respectively. A bit line select voltage(BL_Select) can be applied to each bit line to select that bit line. Forease of explanation, it will be assumed that BL_Select is zero volts,such that the voltage across the non-volatile memory cell in each node112 ₁₁, 112 ₁₂, . . . , 112 ₃₄ is the word line voltage.

In some embodiments, the non-volatile memory cells in nodes 112 ₁₁, 112₁₂, . . . , 112 ₃₄ conduct currents i₁₁, i₁₂, . . . , i₃₄, respectively.Each of the currents i₁₁, i₁₂, . . . , i₃₄ is based on the voltageapplied to the corresponding non-volatile memory cell and theconductance of the corresponding non-volatile memory cell in the node.This “memory cell current” may then flow to the bit line connected tothe non-volatile memory cell. The memory cell current can often bedetermined by multiplying the word line voltage by the conductance ofthe non-volatile memory cell.

Stated another way, each non-volatile memory cell current corresponds tothe result of multiplying one of the elements of an input vector by theweight stored in the non-volatile memory cell. So, for example, anon-volatile memory cell in node 112 ₁₁ conducts a current i₁₁ thatcorresponds to the product Vin₁×w₁₁, the non-volatile memory cell innode 112 ₁₂ conducts a current i₁₂ that corresponds to the productVin₂×w₁₂, the non-volatile memory cell in node 112 ₂₃ conducts a currenti₂₃ that corresponds to the product Vin₃×w₂₃, and so on.

Bit lines BL1, BL2, BL3 may conduct bit line currents Iout₁, Iout₂,Iout₃, respectively. Each bit line current can be understood as thesummation of the currents of the memory cells connected to that bitline. For example, bit line current Iout1=i₁₁+i₁₂+i₁₃+i₁₄, bit linecurrent Iout₂=i₂₁+i₂₂+i₂₃+i₂₄, and bit line currentIout₃=i₃₁+i₃₂+i₃₃+i₃₄. Thus, each bit line current Iout₁, Iout₂, Iout₃may be viewed as representing a sum of products of the input vector withcorresponding weights in a column of the n×m array of weights:

The magnitudes of bit line currents Iout₁, Iout₂ and Iout₃ mayconstitute elements of an output vector and correspond to the associatedvalues of output neurons y₁, y₂ and y₃, respectively. This can thusconstitute the result of a matrix-vector multiplication operation suchas the one depicted in FIG. 1B.

Referring to FIG. 2 , a conceptual cross-point memory array 200 that maybe utilized within a low-power fast-response machine learning systemaccording to various embodiments of the disclosure is shown. In manyembodiments, the cross-point memory array 200 depicted in FIG. 2 may beutilized to perform matrix-vector multiplication operations such asthose depicted in FIG. 1B. Often, the cross-point memory array can beconfigured for use within a memory system and/or a specializedprocessing unit. Various embodiments of the disclosure herein utilize across-point memory array 200 for use in low-power, fast-response machinelearning systems for the generation of a plurality of inferencesregarding image data in a short amount of time.

Cross-point memory array 200 can include n rows and m columns of nodes202 ₁₁, 202 ₁₂, . . . , 202 _(mn). In most embodiments, each of thenodes 202 ₁₁, 202 ₁₂, . . . , 202 _(mn) can include a correspondingnon-volatile memory cell S′₁₁, S′₁₂, . . . , S′_(mn), respectively. Inother embodiments, the cross-point memory array 200 may include morethan one non-volatile memory cell per node.

Each row of nodes 202 ₁₁, 202₁₂, . . . , 202 _(mn) may be coupled to oneof n first conductive lines 206, also referred to herein as word linesWL1, WL2, . . . , WLn 204. For example, in the embodiment depicted inFIG. 2 , the row of nodes 202 ₁₁, 202 ₂₁, 202 ₃₁, . . . , 202 _(m1) iscoupled to word line WL1, the row of nodes 202 ₁₃, 202 ₂₃, 202 ₃₃, . . ., 202 _(m3) is coupled to word line WL3, and so on.

In further embodiments, each column of nodes 202 ₁₁, 202₁₂, . . . , 202_(mn) may also be coupled to one of m second conductive lines 206, alsoreferred to herein as bit lines BL1, BL2, . . . , BLm. For example, asdepicted in FIG. 2 , the column of nodes 202 ₁₁, 202 ₁₂, 202 ₁₃, . . . ,202 _(1n) is coupled to bit line BL1, the column of nodes 202 ₂₁, 202₂₂, 202 ₂₃, . . . , 202 _(2n), is coupled to bit line BL2, and so on.

Each non-volatile memory cell S′₁₁, S′₁₂, . . . , S′_(mn), can beconfigured with a first terminal A₁₁, A₁₂, . . . , A_(mn), respectively,coupled to one of the n word lines WL1, WL2, . . . , WLn, and a secondterminal B₁₁, B₁₂, . . . , B_(mn), respectively, which is furthercoupled to one of the m bit lines BL1, BL2, . . . , BLm. To simplifythis disclosure and to avoid overcrowding the diagram, access devicesare not depicted in FIG. 2 .

In a number of embodiments, each non-volatile memory cell S′₁₁, S′₁₂, .. . , S_(mn), is an Spin Orbit Torque (SOT) MRAM non-volatile memorycell. Low-power, fast-response machine learning techniques that can beutilized in accordance with embodiments of the disclosure are describedin U.S. application Ser. Nos. 17/172,155, 17/172,175, and 17/172,190,which are hereby incorporated by reference in their entirety. In variousembodiments, and as outlined in the above referenced relatedapplications, it is contemplated that other configurations ofcross-point memory arrays may be utilized. For example, the cross-pointarray 200 depicted in FIG. 2 utilizes two-terminal SOT MRAM non-volatilememory cells while other configurations may utilize three-terminalcells.

In many embodiments, the cross-point memory array 200 can operate in aprogramming phase (for programming) and inferencing phase (forgenerating inferences). During the programming phase, each SOT MRAMnon-volatile memory cell S′₁₁, S′₁₂, . . . , S′_(mn) can be programmedto store a corresponding weight of an n×m array of weights w₁₁, w₁₂,w₁₃, W_(nm), respectively. In particular, each SOT MRAM non-volatilememory cell S′_(xx) is often programmed by applying electrical currentpulses from first terminal A_(xx), to second terminal B_(xx). Bothprogramming and inferencing phases can run current pulses from firstterminal A_(xx) to second terminal B_(xx), but programming typicallyruns higher current than inferencing.

During inferencing, SOT MRAM non-volatile memory cells S′₁₁, S′₁₂, . . ., S′_(mn), of cross-point memory array 200 can be operated as describedwithin the above related applications. In particular, during theinferencing phase each SOT MRAM non-volatile memory cell S′₁₁, S′₁₂, . .. , S′_(mn), conducts a memory cell current that corresponds to theresult of multiplying one of the elements of the n-element input vector(multiply vector) by the corresponding weight stored in the non-volatilememory cell.

For example, SOT MRAM non-volatile memory cell S′₁₁ can conduct a memorycell current that corresponds to the product Vin₁×w₁₁, while SOT MRAMnon-volatile memory cell S′₁₂ conducts a memory cell current thatcorresponds to the product Vin₂×w₁₂, and SOT MRAM non-volatile memorycell S′₂₃ conducts a memory cell current that corresponds to the productVin₃×w₂₃, and so on.

During the inferencing phase, the memory cell currents in SOT MRAMnon-volatile memory cells S′₁₁, S′₁₂, . . . , S′_(mn) can flow to thebit line BL1, BL2, . . . , BLm connected to the memory cell. Bit linesBL1, BL2, . . . , BLm may conduct bit line currents Iout₁, Iout₂, . . ., Iout_(m), respectively. Each bit line current is typically thesummation of the memory cell currents of the memory cells connected tothat bit line.

In the embodiments described above, cross-point memory arrays such ascross-point memory array 200 (FIG. 2 ) have been used to implement asingle layer of an artificial neural network 200 that includes inputneurons x₁, x₂, x₃, . . . , x_(n) output neurons y₁, y₂, y₃, . . . ,y_(m), and synapses as nodes 202 that connect input neurons x₁, x₂, x₃,. . . , x_(n) to output neurons y₁, y₂, y₃, . . . , y_(m). It iscontemplated that multi-layer artificial neural networks may beimplemented by cascading cross-point memory arrays so that an output ofa first cross-point memory array can be used as one or more inputs to asecond cross-point memory array, and so on.

In addition, in the embodiments described above, cross-point memoryarray 200 (FIG. 2 ) has been described as being configured to implementa binary neural network in which each SOT MRAM non-volatile memory cellin the array stores a binary weight, n binary inputs are applied to thefirst conductive lines, and m binary outputs are generated at the secondconductive lines. It is further contemplated and would be understood byone skilled in the art that additional circuitry may be used to performoperations such as shift and add to achieve multiple-bit capabilitiesfor the weight matrix for higher precision.

Without being bound by any particular scale, it is believed thatembodiments of the cross-point memory arrays described above may achieverelatively fast speed as a result of various factors including, but notlimited to, parallel in-memory computing without moving data between aprocessor and memory. Additionally, it is believed that many embodimentsof the cross-point memory arrays described above may achieve relativelylow power consumption due to the non-volatile memory nature of MRAM-likeelements. This relatively low-power and fast-response can allow forunique and novel applications and improvements to computing devices andtheir associated technological field. For example, the following figuresdepict how these embodiments may be utilized to generate imagecalibration data when receiving and processing image data such as, butnot limited to, pictures taken with a mobile computing device.

Referring to FIG. 3 , the conceptual process of generating optimizedimage data in accordance with an embodiment of the disclosure is shown.As discussed in more detail in the subsequent discussion of FIGS. 4-8 ,many embodiments of the disclosure utilize a machine learning basedoptimization loop to generate optimized images. An example of thisoptimization loop is depicted in FIG. 3 in response to a user 310 takinga picture with a mobile computing device.

In response to the user 310 initiating the taking of a picture on themobile computing device, an image sensor or image sensor array withinthe device can be directed to take a series of image data 320 which canrepresent various aspects of the image to be captured. Depending on thenumber and/or types of sensors available in the mobile computing device,a series of image data captured can include, but is not limited to, anauto exposed image, an auto white balance image, an autofocused image, anoise reduced image, a local tone mapped image, a highlighted detailsimage, a fused image, a face detected image, a facial landmarked image,a segmented image, and/or a depth image. In certain embodiments, one ormore of the series of image data 320 can be generated based on othercaptured image data which may be internally generated by the imagesensor prior to passing to the memory and/or processor of the mobilecomputing device. In further embodiments, the mobile computing devicemay have a specialized system or logic to pre-process received imagedata to deliver additionally processed image data to the processorand/or memory for further processing.

Upon generation of the series of image data 320, one or more logicswithin the mobile computing device can determine if the image has beenoptimized 330. In response to the determination that the image data isoptimized, the loop can end. In many embodiments described herein, thedetermination of optimization is based upon the analysis of one or moreareas of the image data that are associated with one or more knownsubjects. When the captured series of image data 320 is not optimized,the image data can be processed with machine learning logic 340. Variousembodiments of machine learning processing are described in thesubsequent discussion of FIGS. 4-8 .

In many embodiments, the machine learning logic can analyze the imagedata to determine if a known subject is within the image data. The imagedata may be out of focus on the known subject, which results in anon-optimized image. The machine learning logic can then generate one ormore image sensor configuration changes that can be directed to theimage sensor for capturing more image data utilizing the machinelearning generated configuration changes. As described in more detailbelow, the number and type of configuration changes can depend on thetype of image sensors and other data generating sensors available to themachine learning logic. This image capture, analysis, andreconfiguration loop can repeat for a particular number of cycles, orend upon the capture of an optimized image.

Referring to FIG. 4 , a conceptual schematic diagram of an imageprocessing device 400 suitable for processing optimized image datautilizing low-power fast-response machine learning models in accordancewith an embodiment of the disclosure is shown. Various embodiments ofthe image processing device 400 can include components such as, but notlimited to, a processor 410, an input and output module 420 which mayinclude a plurality of data input and output pathways, an image sensor430, a motion sensor 440, a memory-based machine learning (ML) processor450, and a storage 460. The memory-based ML processor 450 could be basedon the cross-point memory array designs shown above, and could includeassociated control circuitry and analog-to-digital (ADC) to convertreceived digital signal into the analog domain for processing in thememory array, and digital-to-analog (DAC) to convert analog output fromthe memory array back into the digital domain. The memory-based MLprocessor 450 could be configured to implement a plurality of logics 451and 453-454 while the storage 460 is configured to store a plurality ofdata 461-465.

In a number of embodiments, the memory-based ML processor 450 is part ofand controlled by a machine learning logic 452 that can utilize one ormore fast-response, low-power artificial neural networks to generate aplurality of inferences in a manner similar to the discussions of FIGS.1A-2 above. In certain embodiments, the machine learning logic 452 canalso include hardware and/or software logic to format, parse, orotherwise input data into the cross-point memory array of thememory-based ML processor 450 and to process the resultant output. Instill additional embodiments, the machine learning logic 452 may beimplemented as a specialized system on a chip SoC that is configured forspecialized processing of image data.

The image sensor 430 may be configured in a variety of ways based on theapplication desired. In certain embodiments, the image sensor 430 may bean image sensor array comprising multiple lens and image sensor types.The image sensor 430 may be preinstalled image sensors on mobilecomputing devices. By way of example, the image sensor 430 can include awide lens, an ultra-wide lens, a telephoto lens, and a light detectionand ranging (LiDAR) camera. The various lenses may be provided on asingle image sensor or may be disposed individually on separate imagesensors within an image sensor array. In some embodiments, the imagesensor 430 can be externally or remotely connected to the imageprocessing device 400.

In many embodiments, an image processing logic 451 can guide the imageprocessing process through various steps. By way of example, the imagesensor 430 can provide image data, or a series of image data to thememory-based machine learning (ML) processor 450 for processing. Theimage processing logic 451 can pass and/or direct the received imagedata to a machine learning logic 452 for the generation of a pluralityof inferences, to the known subject logic 453 for determining whetherone or more known subjects are within the image data. In variousembodiments, the image processing logic 451 can determine if thereceived image data is optimized, and when optimized, store the imagedata 461 within the storage 460.

In various embodiments, known subject logic 453 may be configured toutilize any known subject data 462 for recognizing a plurality of knownsubjects within a set of image data received for processing. The knownsubject logic 453 may be configured to create new known subject data 462by receiving input data from a user indicating a particular subjectwithin a series of image data is a known subject. In more embodiments,the known subject logic 453 may be directed to comb through or otherwiseaccess a data store of pre-processed images that may allow for thedetermination of one or more known subjects. For example, a user mayallow the known subject logic 453 access to a camera roll of a mobilecomputing device which may have multiple pictures of subjects known tothe user. The known subject logic 453 may be able to determine thefrequency of these subjects and generate new known subject data 462 thatcan be stored for future image data processing.

In further embodiments, image sensor configuration logic 454 can beutilized to provide one or more image sensor configurations to the imagesensor 430. The image sensor 430 may be configurable through one or moresettings. For example, shutter speed, focus, focal length, timings, datatransfer speeds, etc. may all be configured within the image sensor 430to change the image data captured. In various embodiments, the one ormore configurations may be issued to the image sensor 430 to capturemore optimized image data. The image sensor configuration logic 454 canprovide the one or more configurations to the image sensor 430 tofacilitate these subsequent captures.

In various embodiments, the image sensor configuration logic 454 cantranslate inference data 463 generated by the machine learning logic 452to generate image optimization data 465 which can include one or moreimage sensor configurations. In certain embodiments, the imageoptimization data 465 can include only image sensor configurations. Infurther embodiments, the image sensor configuration logic 454 may alsoprocess motion data 464 to generate image optimization data 465. Byutilizing the motion data 464, the image sensor configuration logic 454can generate image sensor configurations that account for motionoccurring to the image sensor 430 and thereby compensate for that asneeded. In some embodiments, the processing of the image sensorconfiguration logic 454 may accomplished by the image processing logic451 or another logic within the image processing device 400, such asprocessor 410. The processor 410 may also be used to implement part orall of image processing logic 451 and known subject logic 453.

In most embodiments, image data 461 can include any data that iscaptured and/or provided by the image sensor 430. Image data 461 canalso include optimized image data that has been stored upon thecompletion of the optimization loop as described in FIG. 3 . Image data461 can be standard pixel color data, but may include a variety of datathat can be captured by an image sensor 430 and may also includemetadata associated with data captured by an image sensor 430 orprocessed by the image processing device 400. As those skilled in theart will recognize, the image data 461 may be located in a storage 460within the image processing device 400 or may be stored within aremovable and/or remote (i.e., “cloud-based”) storage system.

In additional embodiments, known subject data 462 may include data thatallows for the recognition of known subjects within image data 461 thatis being processed. In some embodiments, the known subject data may bepixel data of various known subjects at differing angles, but mayinclude other identifying data which may include, but is not limited to,segmentation data, or facial landmark data. As those skilled in the artwill recognize, known subjects may not be limited to people and may beanimals or other subjects that may be of interest to the user whencapturing images.

In a number of embodiments, inference data 463 can be the output fromone or more machine learning models processed by the machine learninglogic 452. As discussed above, a fast-response, low-power machinelearning system can be utilized to generate a plurality of inferencesthat can be grouped as inference data 463 for further processing.Inference data may be comprised of data that indicates the presence of aknown subject within image data 461 or if the image data 461 isoptimized, in-focus, etc. In some embodiments, the inference data 463generated by the machine learning logic 452 may be immediately processedby another logic to generate further image optimization data 465 withoutbeing directly stored within the storage 460.

In still further embodiments, motion data 464 may indicate the currentmotion of the image sensor 430 or image processing device 400. In manyembodiments, the motion data 464 is directly generated from a motionsensor 440 located within the image processing device 400 oradjacent/associated with the image sensor 430. Utilizing the motion data464, the image processing logic 451 and/or the image sensorconfiguration logic 454 may account for movement during the imagecapture process which may inhibit the optimization of the image data461. For example, the user may be moving the image processing device 400(such as a mobile computing device) during image capture. This maycreate a blur or other distortions in the image data 461 being captured.The motion data 464 may be added to the inference data 463 in certainembodiments to generate image optimization data 465 which can attempt tocompensate for this motion by issuing or modifying one or more imagesensor configurations. Similar to other data 461-463, 465 the motiondata 464 may not be stored directly within the storage 460 of the imageprocessing device 400 but may be directly generated and processed by oneor more logics 451, and 453-454 before being deleted.

In some embodiments, image optimization data 465 can include datagenerated upon parsing inference data 463 generated from the machinelearning logic 452. In a variety of embodiments, the image optimizationdata 465 is a data processed by the image sensor configuration logic 454which includes one or more image sensor configurations. As described inmore detail below, the image optimization data 465 can include imagesensor configurations which are generated to optimize one or more knownsubject areas within the image data 461. In some embodiments, the imageoptimization data 465 can be generated based on a combination ofinference data 463 and motion data 464. The image optimization data 465can often be parsed and processed by the image sensor 430 to capturesubsequent image data 461. In other embodiments, the image optimizationdata 465 requires one or more logics to translate and/or issue commandsto the image sensor 430.

Referring to FIG. 5A, a conceptual illustration of an image 510 with aknown subject 520 processed with a bounding box 540 in accordance withan embodiment of the disclosure is shown. To illustrate variousembodiments described herein, FIG. 5A depicts a sample image taken oftwo dogs on a landscape. It may be the case that the left dog is a knownsubject 520 while the dog on the right is an unknown subject 530. Theselection of a known subject can be represented by generating a boundingbox 540 around the known subject 520 within the image 510. The boundingbox 540 can be understood to be the smallest possible rectangle thatencases the entire known subject 520. However, in certain embodiments,the bounding box 540 may be limited to a facial/bust area of a subjectto maximize known subject detection. The bounding box 540 can then be anarea selected for processing to determine if the image 510 has beenoptimized.

In certain embodiments, an unknown subject 530 may be a subject notknown to the user, or may be a subject that has not either beenexplicitly selected for recognition by the user or has not beenphotographed enough with the image taking device to exceed apredetermined threshold of recognition. In additional embodiments, thedetermination of a known subject 520 within an image 510 can be based onpreviously stored known subject data within the image capturing device,such as a mobile computing device. In further embodiments, the imagecapturing device may have access to a cache of previously capturedimages that may serve as a source of determining the presence of knownsubjects or for the creation of a database of known subjects. Theselection of known subjects may be generated by the presence of thesubject within a predetermined number of pictures, or may be generatedby the presence of a sufficient amount of data required to create aknown subject profile. In still more embodiments, the determination of aknown subject 520 may be manually accomplished by a user, sometimes inresponse to one or more prompts that indicate whether a selected subjectis known or should be classified as known.

Referring to FIG. 5B, a conceptual illustration of an image 550 with aknown subject 520 processed with a pixel mask 560 in accordance with anembodiment of the disclosure is shown. As described above with referenceto FIG. 5A, an image may comprise a known subject 520 and an unknownsubject 530. The selection of the area that encases a known subject 520for purposes of image optimization may occur in a variety of ways. FIG.5A depicted a bounding box 540 that encased the full known subject 520.The embodiment depicted in FIG. 5B utilizes a pixel mask 560 that coversthe entire known subject 520 on an exact, or near-exact pixel-level. Inthis sense, a mask is generated that closely matches the known subject.In this way, only the area of the known subject 520 is selected and isutilized to determine image optimization. Benefits of utilizing a pixelmask are more accurate optimizations, however the disadvantages includeincreased processing time and power to generate the pixel mask 560compared to a bounding box 540.

Referring to FIG. 6 , a conceptual illustration of an image 610processed with multiple known subjects 620, 630 in accordance with anembodiment of the disclosure is shown. In a variety of embodiments,images may be captured that include multiple known subjects and unknownsubjects. It may be the desire of a user to want an image 610 to becaptured of the known subjects 620, 630, at the potential expense of theunknown subjects 640. In these embodiments, the image capturing devicemay generate two bounding boxes 650, 660 that encase a first knownsubject 620 and second known subject 630. As described above, thebounding boxes 650, 660 depicted in FIG. 6 are limited to thefacial/bust region of the known subjects 620, 630. In this way, theoptimization of the image 610 can be limited to their faces. It shouldbe noted that although an image 610 may be taken and optimized based onthe data associated with the one or more known subjects 620, 630, thatdoesn't automatically mean such optimization is done at the expense ofthe unknown subjects 640.

In many embodiments, images can be captured that may provide clarity inone portion of the image while sacrificing the quality of the image inanother location of the frame. By determining and selecting one or moreknown subjects 620, 630 and generating a bounding boxed 650, 660associated with them, can allow the image capturing system to be guidedby a preference that if such quality decisions and/or tradeoffs are tobe made while capturing the image, that those decisions or processingshould focus or benefit the data within the known subjects boundingboxes 650, 660 over data outside. These determinations and preferencescan further be utilized to continue the image capture process until apredetermined level of optimization is achieved. Although the image 610depicted in FIG. 6 shows two known subjects, it is contemplated thatimages may be captured with additional known subjects which is onlylimited by the processing power available and/or the amount of imagespace utilized.

Referring to FIG. 7 , a flowchart depicting a process 700 for optimizingimage data with a plurality of known subjects in accordance with anembodiment of the disclosure is shown. In many embodiments, image datais received for processing (block 710). The image data may be receivedby an image sensor or image sensor array. In certain embodiments, theimage data may be received from a data file or from a remotely locatedimage sensor. The received image data can be processed by one or morelow-power, fast-response machine learning systems (block 720). Asdescribed above, the low-power, fast-response machine learning systemcan be an artificial neural network or other cross-point memory arraysuch as those described above with respect to the discussion regardingFIGS. 1A-2 .

The one or more machine learning systems can generate a plurality ofinference data that can be utilized to recognize a plurality of knownsubjects within the received image data (block 730). In response to therecognition of a plurality of known subjects, the process 700 can inresponse select a known subject area within the received image data. Asdiscussed in FIGS. 5A-5B, the known subject area can be processed as aboundary box, pixel mask, or other similar demarcations. In response toa selection being made, the process 700 can determine if all of therecognized subjects have been selected within the image data (block745). When one or more recognized subjects have not been selected withinthe image data, the process 700 can again select another known subjectarea (block 740). When all recognized subjects have been selected, theprocess 700 can optimize the newly received image data throughgenerating one or more image sensor configuration changes based on theareas of the selected known subjects (block 750). Once optimized, theimage data can be stored within one or more storage devices (block 760).

Referring to FIG. 8 , a flowchart depicting a process 800 for utilizingfast-response low-power machine learning logic to optimize receivedimage data in accordance with an embodiment of the disclosure is shown.In many embodiments, the process 800 can commence upon receiving imagedata from an image sensor (block 810). As discussed previously, theimage sensor may be configured within a mobile computing deviceincluding, but not limited to, a smart phone and/or a mobile tablet. Thereceived image data can be processed to detect if one or more knownsubjects are within the image data (block 820). The process 800 can thendetermine if any know subjects are recognized within the processed image(block 825). If no known subjects are recognized within the image data,the process 800 can utilize one or more conventional autofocus methodsto optimize the image data (block 830). As those skilled in the art willrecognize, conventional autofocus methods may include, but are notlimited to, phase and/or contrast detection method. Once optimized, theimage data can then be stored (block 840).

In a variety of embodiments, when the process 800 determines that one ormore known subjects are recognized within the processed image data, theimage data is passed to a low-power, fast-response machine learninglogic (block 850). the low-power, fast-response machine learning logiccan be similar to those described above with respect to FIGS. 1-2 and 4. In a number of embodiments, the low-power, fast-response machinelearning logic can generate a plurality of image enhancement inferencesvia the machine learning logic (block 860). In other words, theplurality of inferences can be utilized to optimize the image data. Inmost embodiments, the inferences are further configured to optimize theimage data within one or more regions within the image data thatcorresponds to one or more of the known subjects that have beenrecognized. In certain embodiments, the low-power, fast-response machinelearning logic may also be utilized to perform the subject recognition,autofocusing, and/or the image area selection steps of the process 800.

Once the inferences are generated, the process 800 can determine if thephotograph within the image data is sufficiently optimized (block 865).It is contemplated that image data may comprise more than just aphotograph. This can include, but is not limited to, metadata, depthmaps, supplemental image channels, etc. If it is determined that theimage data is sufficiently optimized, the process 800 can then store theimage data (block 845). In a variety of embodiments, the determinationof whether image data is sufficiently optimized occurs prior to thegeneration of inferences from the low-power, fast-response machinelearning logic.

In response to the determination that the image data and/or photographis not sufficiently optimized, the process 800 can further determine ifa motion sensor detects image sensor movement (block 875). As can beunderstood by those skilled in the art, an image sensor disposed withina mobile computing device can be moved about during the process ofcapturing image data. This movement may change the potential foroptimizing an image during capture. To compensate for this potentialmovement, various changes to an image sensor may be made. In response tothe determination is that motion is occurring during the image captureprocess by receiving motion sensor data, the process 800 can generateimage optimization data based on the plurality of image enhancementinference and detected motion sensor data (block 890). Based upon bothtypes of data, the generated image optimization data can be utilized tochange one or more image sensor configurations (block 895).

When the process 800 determines that a motion sensor does not detectimage sensor movement, the generation of image optimization data can bebased solely on the plurality of image enhancement inferences (block880). It is contemplated that further embodiments may utilize othersupplemental types of data for generating image optimization data beyondmotion data. Similarly, the process 800 can change one or more imagesensor configurations based on the image optimization data generatedfrom the plurality of image enhancement inferences (block 895).

Upon changing the one or more sensor configurations, further image datacan be captured. The process 800 can then wait from subsequent imagedata to be received from the changed image sensor (block 810). In manyembodiments, this cycle can repeat until it is determined that the imagedata being captured is sufficiently optimized and the image data isstored within one or more storage devices (blocks 865, 840).

Information as herein shown and described in detail is fully capable ofattaining the above-described object of the present disclosure, thepresently preferred embodiment of the present disclosure, and is, thus,representative of the subject matter that is broadly contemplated by thepresent disclosure. The scope of the present disclosure fullyencompasses other embodiments that might become obvious to those skilledin the art, and is to be limited, accordingly, by nothing other than theappended claims. Any reference to an element being made in the singularis not intended to mean “one and only one” unless explicitly so stated,but rather “one or more.” All structural and functional equivalents tothe elements of the above-described preferred embodiment and additionalembodiments as regarded by those of ordinary skill in the art are herebyexpressly incorporated by reference and are intended to be encompassedby the present claims.

Moreover, no requirement exists for a system or method to address eachand every problem sought to be resolved by the present disclosure, forsolutions to such problems to be encompassed by the present claims.Furthermore, no element, component, or method step in the presentdisclosure is intended to be dedicated to the public regardless ofwhether the element, component, or method step is explicitly recited inthe claims. Various changes and modifications in form, material,work-piece, and fabrication material detail can be made, withoutdeparting from the spirit and scope of the present disclosure, as setforth in the appended claims, as might be apparent to those of ordinaryskill in the art, are also encompassed by the present disclosure.

What is claimed is:
 1. A device, comprising: a memory; and a processorcommunicatively coupled to the memory, the processor being configured todirect the device to: receive image data; pass the received image datato a machine learning model; recognize a known subject within the imagedata; generate a plurality of inferences via the machine learning model;utilize the plurality of inferences to optimize a portion of the imagedata associated with the known subject during subsequent image datacaptures.
 2. The device of claim 1, wherein the processor comprises amachine learning processor comprising a plurality of cells configured tostore weights for the machine learning model.
 3. The device of claim 2,wherein the plurality of inferences are generated by applying signalsassociated with the received image data to the plurality of cells viaone or more signal lines associated with the memory cells.
 4. The deviceof claim 1, wherein the device is further configured to: receive motiondata from a motion sensor; and optimize the portion of the image databased on the plurality of inferences and the received motion data. 5.The device of claim 1, wherein the image data is captured in a series.6. The device of claim 5, wherein the series of captured image datarepresents various aspects of the image to be captured.
 7. The device ofclaim 6, wherein the various aspects can include any of: an auto exposedimage, an auto white balance image, an autofocused image, a noisereduced image, a local tone mapped image, a highlighted details image, afused image, a face detected image, a facial landmarked image, asegmented image, or a depth image.
 8. The device of claim 6, wherein thedevice generates unique image optimization data for each various aspectof the series of captured image data.
 9. The device of claim 1, whereinthe device is configured to continue processing image data until theoptimization of the image data exceeds a predetermined threshold. 10.The device of claim 9, wherein the predetermined threshold is related tothe focus of the known subject.
 11. The device of claim 1, wherein thedevice is further configured to recognize two or more known subjectswithin the image data and optimize two or more discrete portions of theimage data corresponding to each of the two or more known subjects. 12.A method for generating image optimization data, comprising: receivingimage data; processing the received image data, wherein the processingincludes: passing the received image data to machine learning model;recognizing one or more known subjects within the image data;determining an area for optimization based on the one or more knownsubjects; generating a plurality of inferences via the machine learningmodel; utilizing the plurality of inferences to generate one or moreimage sensor configurations for optimizing the determined area withinthe image data; and providing the one or more image sensorconfigurations to an image sensor for capturing subsequent image data.13. The method of claim 12, wherein the method further includesdetermining if the received image data exceeds a predeterminedoptimization threshold prior to processing.
 14. The method of claim 13,wherein the method further comprises receiving and processing subsequentimage data until the predetermined optimization threshold is exceeded.15. The method of claim 12, wherein the optimization includes improvingthe focus of the determined area.
 16. A device, comprising: an imagesensor; a memory for storing data and executable instructions; and aprocessor communicatively coupled to the memory, the processor beingconfigured to direct the device to: receive image data; pass thereceived image data to a machine learning model; recognize a knownsubject within the image data; generate a plurality of inferences viathe machine learning model; utilize the plurality of inferences togenerate image sensor configurations for optimizing image data captureareas associated with the known subject.
 17. The device of claim 16,further comprises an image sensor array, wherein the image sensor arrayis disposed within a mobile computing device.
 18. The device of claim17, wherein the image sensor array is configured to capture a series ofimage data.
 19. The device of claim 18, wherein image sensorconfigurations are generated for each image data of the series of imagedata.
 20. The device of claim 16, wherein the known subject is selectedvia an input received by the mobile computing device.